As we continue to develop CMOS implementations of the widely employed ESA/390 multi-processor system we are faced with the need to improve performance with a compatible design. The ESA/390 architecture includes several operations which must appear to occur at a single point in time as seen by all CPUs in a multi-processor system. These operations are primarily used to update storage access control information, which may in general be buffered within the CPUs to provide high-performance access to this information. That buffered information may in many cases be referenced multiple times within the execution of a single ESA/390 instruction on one CPU, and the ESA/390 architecture requires that certain changes to that information not appear to happen within an ESA/390 instruction.
The two operations which are used frequently enough to affect system performance are Invalidate Page Table Entry (IPTE) and Set Storage Key Extended (SSKE) and in order to improve performance we have developed a way of handling these operations more efficiently and thus can reduce system runtime of programs which employ these instructions. In the course of executing one of these instructions on a single CPU, the Dynamic Address Translation (DAT) path to a 4K block of virtual storage is made invalid (IPTE) or the storage key used to control access to a 4K block of real storage is changed (SSKE). These changes must be transmitted to all CPUs in the system as part of the execution of the instruction on the originating CPU.
A straightforward implementation of the ESA/390 architecture for these instructions is for the CPU executing the IPTE or SSKE to signal all other CPUs in the system to pause at a point between ESA/390 instructions, a quiesce point in the usual quiesce method used in the S/370 based systems. Once all CPUs have reached such a point, the system is said to be quiesced and the first CPU is free to update the translation table entry or storage key as needed. While the system is quiesced, the first CPU must also indicate the change being made to the other CPUs so that those other CPUs, along with the first CPU, can update or discard any buffered copies of the information being changed. Once all of these actions are complete, the first CPU signals the other CPUs that they may resume processing. This approach has been used on a number of S/370, 370/ESA, and ESA/390 implementations and thus may be called a `quiesce method`.
Here, as an aside, we should note that a recent ESA/390 implementation was described in U.S. Ser. No. 08/689,762, filed Aug. 13, 1996, issued as U.S. Pat. No. 5,761,734 on Jun. 2, 1998 entitled "Token-based Serialization of Instructions in a Multiprocessor System" by inventors, H. W. Tast, U. Gaertner, K. J. Getziaff, and E. Pfeffer, assigned to International Business Machines Corporation, the assignee of this application. The invention described was used in an ESA/390 processor introduced commercially late in 1996 when performing instructions like IPTE (Invalidate Page-Table Entry) and SSKE (Set Storage Key Extended which modify the Tables in Memory that are commonly used by all the processors in order to perform address translations was described. There a token was used for serializing instructions like IPTE and SSKE which have had to be processed serially. After a request in response to a request of one of the processors the token is assigned to the requesting process which then has the right to execute one of said instructions. In case the instruction comprises several tasks, the token is locked until the last task of the instruction is executed. Then, the token is unlocked and can be assigned to another instruction. A processor can only execute one of the instructions that have to be processed serially if it possesses a token. When there is a need for serializing instructions that have to be processed serially in a multiprocessor system with this token apparatus, a first task can be only initiated on an initiating processor which can only execute the first task if it possesses a token, and in order to have a token assigned to one processor in case it is available, there must first be a request for the token by the initiating processor, and when it is available, the instruction can be executed serially, after which there is a need for returning the token after completing the first task of an instruction that has to be processed serially, with the token not necessarily becoming available for other processors. After completion of a serial task there must be a reestablishment of the availability of the token after completion of the instruction that has been processed serially.
A drawback to the required steps is that with the control of the process by the initiating processor and the complexity of token passing, there can be no system serialization with early release of individual processors in the ESA/390 system in the recently released CMOS processor version. While it would be possible to store the contents of an entry of an address translation table in separate registers and then invalidate the entry, until now, for the IPTE and SSKE instructions, the commercially implemented recognized alternative to the token process is when modifying an address translation table to wait until all processors have finished the execution until all processors have finished their current instruction, the quiesce method usually used in the other described System 370 based system, including earlier versions of the ESA/390 systems.
A chief drawback to waiting for all CPUs to quiesce (the quiesce method) is that the time required for a CPU to reach a point that is between ESA/390 instructions is highly variable due to the presence of complex and long-running instructions in the ESA/390 instruction set.
In a system with a large number of CPUs, the mean time until all CPUs in the system have reached such a point can grow large, and the net effect on system performance is compounded by the fact that all CPUs are waiting (doing no useful work) until the last CPU reaches such a point. This, and the amount of handshaking required among the CPUs, makes the performance of such an approach unattractive.
Another approach that has been used (e.g. on the ES/9000 9021) is to have the CPU executing the IPTE or SSKE first change the translation table or storage key, then signal all other CPUs to update or discard their buffered copies as soon as they reach a point between ESA/390 instructions. This is possible only if the CPU design is such that ESA/390 instructions can be completely nullified at any point prior to instruction completion. This requirement stems from the fact that the change to the translation table or storage key could cause a storage access exception condition to become apparent to another CPU before it has reached the end of the ESA/390 instruction it is currently executing. If in this case something else causes that CPU's buffered copy of the changing information to be discarded (e.g. replacement in a set-associative lookaside buffer) during that instruction, the CPU may see an access exception on an operand it had previously found to be accessible. If the ESA/390 instruction can still be nullified at this point, all is well, but if not then the CPU operation fails to satisfy the ESA/390 architecture. This approach has considerably better performance than the straightforward approach, but carries the requirement of nullifiability throughout the execution of ESA/390 instructions. In a CPU design that has this feature anyway (e.g. to support speculative or out-of-sequence execution), this is a reasonable design for IPTE and SSKE; the performance benefit for these instructions alone, however, could not justify adding this degree of complexity to a CPU design.
A form of the latter approach is also possible if the CPU design can guarantee that buffered copies of the relevant information cannot be lost during the course of an ESA/390 instruction. While it is not difficult to make this a rare event, making it impossible places a significant constraint on the CPU design, particularly given the ESA/390 Interpretive Execution architecture, which in some cases requires recursive DAT and allows intermediate translations to be kept in a Translation Lookaside Buffer (TLB).
An implementation of ESA/390 IPTE and SSKE operations is therefore desired which does not have the burdensome CPU design requirements of either of the latter two implementations, but which does not incur the performance overhead of the first implementation.